Qualified command

ABSTRACT

A method for executing a qualified command including detecting a hand gesture input for identifying a command, detecting one or more non-hand gesture inputs to qualify the command, and configuring a processor to execute the qualified command on a machine in response to the hand gesture input and one or more non-hand gesture inputs.

BACKGROUND

When inputting commands to be executed on a machine, a user accesses oneor more input devices to input the command. When one or more inputdevices detect the input, one or more input devices will transferinformation corresponding to the input to the machine. The machine thenexecutes one or more commands in response to the information.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features and advantages of the disclosed embodiments will beapparent from the detailed description which follows, taken inconjunction with the accompanying drawings, which together illustrate,by way of example, features of the embodiments.

FIG. 1 illustrates a machine coupled to one or more cameras and one ormore input device according to an embodiment of the invention.

FIG. 2 illustrates a block diagram of a gesture application qualifying acommand in response to one or more hand gesture inputs and one or morenon-hand gesture inputs according to an embodiment of the invention.

FIG. 3 illustrates a camera detecting a user's hand gesture input and amicrophone detecting a user's non-hand gesture input according to anembodiment of the invention.

FIG. 4 illustrates users entering hand gesture inputs and non-handgesture inputs to qualify commands that will be executed on objectsshown on a display device according to an embodiment of the invention.

FIG. 5 illustrates a machine with an embedded gesture application and agesture application stored on a removable medium being accessed by themachine according to an embodiment of the invention.

FIG. 6 is a flow chart illustrating a method for executing a qualifiedcommand according to an embodiment of the invention.

FIG. 7 is a flow chart illustrating a method for executing a qualifiedcommand according to another embodiment of the invention.

DETAILED DESCRIPTION

FIG. 1 illustrates a machine coupled to one or more cameras 140 and oneor more input device 150 according to an embodiment of the invention. Inone embodiment, the machine 100 is a desktop, a laptop, a server, and/orany device that is coupled to one or more cameras 140 and/or one or moreinput devices 150. As illustrated in FIG. 1, the machine 100 includes aprocessor 120, one or more cameras 140, one or more input devices 150, amemory/storage device 180, a network interface 160, and a communicationbus 170 for the machine 100 and/or one or more components of the machine100 to communicate with one another.

Additionally as illustrated in FIG. 1, the storage device 180 stores agesture application 110, recognized hand gestures 130, and recognizedqualifiers 135. In other embodiments, the machine 100 includesadditional components and/or is coupled to additional components inaddition to and/or in lieu of those noted above and illustrated in FIG.1.

As noted above, the machine 100 includes a processor 120. The processor120 sends data and/or instructions to the components of the machine 100,such as one or more cameras 140, one or more input devices 150, and agesture application 110. Additionally, the processor 120 receives dataand/or instruction from components of the machine 100, such as one ormore cameras 140, one or more input devices 150, and the gestureapplication 110.

The gesture application 110 sends instructions for one or more cameras140 and/or one or more input devices 150 to capture hand gesture inputsand/or non-hand gesture inputs. Additionally, the gesture application110 analyzes the captured hand gesture inputs and identifies a commandthat corresponds to a captured hand gesture input and can be executed bythe machine 100. Further, the gesture application 110 will attempt toqualify the command by analyzing the captured non-hand gesture inputsand identifying one or more qualifiers that correspond to the non-handgesture inputs (the identified command). Once the gesture application110 has qualified the command, the gesture application 110 will instructthe processor 120 to execute the qualified command. The qualifiedcommand can be executed on one or more objects, devices, components,machines and/or environments.

The command is an instruction that can be executed by the machine 100when the gesture application 110 identifies a command corresponding to acaptured hand gesture input. In one embodiment, the command will be tozoom in/out, increase/decrease volume, proceed to next/previous, scrollup/down, open/close, rotate, browse, etc. In other embodiments,additional commands and/or instructions can be executed by the machine100 when the gesture application 110 identifies a hand gesture inaddition to and/or in lieu of those noted above.

Additionally, as noted above, the command can be qualified with one ormore qualifiers. One or more qualifiers define parameters for thecommand. In one embodiment, one or more qualifiers specify degrees ofthe command, indicate which object on a display device to execute thecommand on, and/or specify which user is entering the command. Whenspecifying a degree of the command, one or more qualifiers can indicatehow much to perform a scrolling command, how much to increase/decreasevolume, and/or how many times to zoom in/out. In another embodiment, oneor more qualifiers are speech qualifiers that reinforce the command byrepeating the command as part of the qualifier. In other embodiments,one or more qualifiers can compliment the command in additional waysand/or specify additional parameters in addition to and/or in lieu ofthose noted above. In identifying one or more qualifiers, the gestureapplication 110 will identify one or more non-hand gesture inputs.

Once the command has been qualified, the gesture application 110 willexecute the qualified command. In one embodiment, the gestureapplication 110 is firmware that is embedded onto the machine 100. Inother embodiments, the gesture application 110 is a software applicationstored on the machine 100 within ROM or on a storage device 180accessible by the machine 100 or the gesture application 110 is storedon a computer readable medium readable and accessible by the machine 100from a different location. Additionally, in one embodiment, the storagedevice 180 is included in the machine 100. In other embodiments, thestorage device 180 is not included in the machine, but is accessible tothe machine 100 utilizing a network interface 160 included in themachine 100. The network interface 160 may be a wired or wirelessnetwork interface card.

In a further embodiment, the gesture application 110 is stored and/oraccessed through a server coupled through a local area network or a widearea network. The gesture application 110 communicates with devicesand/or components coupled to the machine 100 physically or wirelesslythrough a communication bus 170 included in or attached to the machine100. In one embodiment the communication bus 170 is a memory bus. Inother embodiments, the communication bus 170 is a data bus.

As noted above, the machine 100 includes one or more cameras 140. One ormore cameras 140 are imaging devices that are instructed by the gestureapplication 110 to concurrently or sequentially capture images and/orvideos of users entering one or more hand gesture inputs and/or one ormore non-hand gesture inputs. In one embodiment, one or more cameras 140are embedded as part of the machine 100. In another embodiment, one ormore cameras 140 are coupled to the machine 100 and are positioned atdifferent locations in an environment where the user is located. Inother embodiments, one or more cameras 140 are worn by the user as headgear or accessories.

Additionally, one or more hand gesture inputs are first modes of inputsthat include one or more motions made by one or more of the user'shands. In one embodiment, the hand gesture inputs additionally includeshapes made by one or more of the user's hands. Further, in making thehand gesture inputs, the user can use a single hand to make one or moremotions and/or one or more shapes. In another embodiment, the user canutilize both hands to make one or more motions and/or one or moreshapes.

In addition, in one embodiment, one or more cameras 140 capture the handgesture inputs as a video stream for the gesture application 110. Inanother embodiment, one or more cameras 140 capture one or more handgesture inputs as multiple still images for the gesture application 110to analyze and identify. In other embodiments, one or more input devices150 can also be utilized individually or in conjunction with one or morecameras 140 to detect and capture the hand gesture inputs.

One or more input devices 150 are devices that can concurrently orsequentially detect and capture hand gesture inputs and/or non-handgesture inputs. Similar to one or more cameras 140, in one embodiment,one or more input devices 150 are embedded as part of the machine 100.In another embodiment, one or more input devices 150 are coupled to themachine 100 and are positioned at different locations in an environmentwhere the user is located. In other embodiments, one or more inputdevices 150 are worn by the user as head gear or accessories.

Additionally, in one embodiment, one or more input devices 150 are amouse or a touch screen that can detect a user's hand gesture input whenthe user is accessing the mouse or touch screen. In another embodiment,one or more input devices 150 are a microphone, a biometric device, oneor more cameras or a motion detection device, such as an accelerometer,tilt sensor, illumination sensor, and/or proximity sensor. The camerasmay be additional cameras or be included as one or more cameras 140.Further, one or more input devices 150 can be used independently or inconjunction with one or more cameras 140 to capture one or more non-handgesture inputs.

One or more non-hand gesture inputs are inputs that are not made by oneor more of the user's hands. The non-hand gesture inputs are second modeof inputs that are used to qualify the command. In one embodiment, oneor more non-hand gesture inputs include audible speech. The audiblespeed may specify a degree of how much to perform the command and/or theaudible speech can reinforce the command by including the commandaudible form. In another embodiment, one or more non-hand gesture inputsinclude a gaze. The gaze corresponds to what direction the user islooking and to what the user is looking at. In other embodiments, thenon-hand gesture inputs include additional forms of input, such as thebiometrics of the user.

Further, in one embodiment, the gesture application 110 can instruct oneor more cameras 140 and/or one or more input devices 150 to capture oneor more hand gesture inputs before capturing one or more non-handgesture inputs. In other embodiments, the gesture application 110instructs one or more cameras 140 and/or one or more input devices 150to capture one or more hand gesture inputs while concurrently capturingone or more non-hand gesture inputs.

Once one or more hand gesture inputs have been captured, the gestureapplication 110 analyzes the hand gesture input to identify acorresponding command. As noted above, the command is an instructionthat can be executed by the machine 100 when the gesture application 110determines that a hand gesture input matches one of the recognized handgestures 130 stored on a storage device 180. In other embodiments, therecognized hand gestures 130 and/or the recognized qualifiers 135 arestored on another device accessible to the machine 100 through anetwork.

The recognized hand gestures 130 are stored hand gestures that themachine 100 recognizes. In one embodiment, each recognized hand gesture130 lists a number of hands used for the corresponding recognized handgesture 130. In another embodiment, each recognized hand gesture 130additionally lists one or more motions associated with the recognizedhand gesture 130. In other embodiments, each recognized hand gesture 130list a shape made by one or more hands.

In analyzing one or more hand gesture inputs, the gesture application110 will analyze captured images and/or video streams from one or morecameras 140. In another embodiment, the gesture application 110 willadditionally analyze data from one or more input devices 150. Thegesture application 110 will analyze the images, video streams, and/ordata to determine one or more motions made by the user when entering thehand gesture input. In one embodiment, the gesture application 110 willadditionally determine from the images, video streams, and/or data howmany hands were used in entering the hand gesture input. In otherembodiments, the gesture application 110 will identify one or moreshapes made by one or more of the user's hands.

Once the gesture application 110 has analyzed one or more hand gestureinputs, the gesture application will scan the recognized hand gestures130 for a matching recognized hand gesture. In one embodiment, thematching recognized hand gesture will list one or more motions matchingone or more motions from the hand gesture input. In another embodiment,the matching recognized hand gesture also lists a number of hands and/orone or more shapes made that match the number of hands and/or the shapeof the hands made by the hand gesture input. Further, the matchingrecognized hand gesture will have a corresponding command that can beexecuted by the machine 100. As a result, after analyzing one or morehand gesture inputs, the gesture application will identify a commandthat corresponds to the corresponding hand gesture input.

Once the gesture application 110 has identified the correspondingcommand for the hand gesture input, the gesture application 110 willattempt to qualify the command using one or more of the non-hand gestureinputs. In one embodiment, if the gesture application does not detectand/or recognize one or more non-hand gesture inputs, the gestureapplication 110 can utilize a default qualifier to qualify the command.In other embodiments, in qualifying the command, the gesture application110 will analyze one or more non-hand gesture inputs to identify one ormore qualifiers.

The gesture application 110 will analyze images, video streams, and/ordata from one or more cameras 140 and/or one or more input devices 150to determine whether the non-hand gesture input includes, speech, gaze,and/or biometrics. The gesture application 110 will then compare thecaptured speech, gaze, and/or biometrics from the non-hand gesture inputand scan one or more recognized qualifiers 135 corresponding to thematching recognized hand gesture.

As illustrated in FIG. 1, each of the recognized hand gestures 130include one or more corresponding recognized qualifiers 135. One or morerecognized qualifiers 135 are qualifiers that the machine 100 recognizesand can be used with the command corresponding to the matching handrecognized hand gesture. In one embodiment, each recognized qualifierlists one or more non-hand gesture inputs that correspond to therecognized qualifier. The gesture application 110 will scan therecognized qualifiers 135 for speeches, gazes, commands, and/orbiometrics that match the non-hand gesture input.

If one or more of the recognized qualifiers 135 have matching speeches,gazes, commands, and/or biometrics, the gesture application 110 proceedto qualify the command with the matching recognized qualifiers 135. Oncethe gesture application 110 has qualified the command with one or morequalifiers, the gesture 110 application can proceed to execute thequalified command on one or more objects, devices, components, machines100 and/or environments.

In one embodiment, the gesture application 110 can also utilize one ormore cameras 140 and/or one or more input devices 150 to identify auser. The gesture application 110 can instruct one or more cameras 140and/or one or more input devices 150 to capture hand gestures, speeches,and/or gazes when identifying the user. The gesture application 110 canutilize hand gestures, gazes, and/or voices that correspond to a userand are stored on the storage device 180. By identifying the user, thegesture application 110 can grant or deny access to the user inexecuting the qualified command on the object, device, machine, and/orenvironment.

FIG. 2 illustrates a block diagram of a gesture application 210qualifying a command 215 in response to one or more hand gesture inputs220 and one or more non-hand gesture inputs 230, 235 according to anembodiment of the invention. As noted above, the gesture application 210utilizes one or more cameras and/or one or more input devices to captureone or more hand gesture inputs 220 and one or more non-hand gestureinputs 230, 235.

As illustrated in FIG. 2, in one embodiment, the gesture application 210will analyze the hand gesture input 220 and determine that the handgesture input 220 indicates that one hand is used, all of the fingers ofthe hand are extended, and that the hand is moving in a rotating manner.The gesture application 210 will then attempt to identify acorresponding command for the hand gesture input 220. As noted above,the gesture application 210 will scan recognized hand gestures 240, 250,and 260 to find a matching recognized hand gesture.

As illustrated in FIG. 2, recognized hand gesture 2 250 lists that onehand is to be used, all of the fingers of the hand are extended, and thehand will move in a rotating manner. As a result, in the presentembodiment, the gesture application 210 will determine that recognizedhand gesture 2 250 is a matching recognized hand gesture. Additionally,the gesture application 210 will identify that the command, “RotateObject,” corresponds to recognized hand gesture 2 250.

Once the gesture application 210 has identified the command, “RotateObject,” the gesture application 210 will proceed to attempt to qualifythe command with non-hand gesture inputs 230, 235. As noted above, oneor more qualifiers may include speech and/or gaze. Additionally, in oneembodiment, one or more qualifiers may specify an object, machine,and/or device to execute the command on. In other embodiments, one ormore qualifiers may additionally specify degrees of the command.

As illustrated in FIG. 2 and as noted above, each of the recognized handgestures include one or more corresponding recognized qualifiers. Asshown in FIG. 2, recognized hand gesture 2 250 includes 3 correspondingrecognized qualifiers, 283, 286, 289. Additionally, as illustrated inFIG. 2, other recognized hand gestures may include a different number ofrecognized qualifiers and different corresponding commands. The gestureapplication 210 will analyze the non-hand gesture inputs 230, 235 toidentify corresponding qualifiers that can be used with the command.

In one embodiment, as illustrated in FIG. 2, the gesture applicationwill determine that non-hand gesture input 1 230 is a speech input. Thegesture application 210 will scan recognized qualifiers 283, 286, and289, which correspond to recognized hand gesture 2 250, to determinewhether any of the recognized qualifiers include speech. The gestureapplication will determine that recognized qualifiers 286 and 289include speech. Additionally, the gesture application 210 goes ontodetermine whether recognized qualifiers 286 or 289 include the speech“Rotate by 30 Degrees.” The gesture application 210 will identify thatrecognized qualifier 286 includes speech to “Rotate N Degrees” anddetermine that recognized speech qualifier 286 can be used to qualifythe command.

Additionally, the gesture application 210 will determine that the number30 specified in the non-hand gesture input 1 230 specifies the amount ordegree of rotation. Further, the gesture application 210 will alsoanalyze non-hand gesture input 2 235 and attempt to qualify the command.The gesture application 210 will determine that non-hand gesture input 2235 does not include a speech input, but it does include a gaze input.The gesture application 210 will read from the non-hand gesture input 2235 that a user entering the hand gesture input 220 and the non-handgesture inputs 230, 235 has a line of sight on object 1.

The gesture application 210 will again scan the recognized qualifiers283, 286, and 289 to determine whether any of the recognized qualifiersinclude a gaze qualifier that can specify an object. As illustrated inFIG. 2, the gesture application will determine that recognized qualifier283 includes a gaze qualifier that can specify an object to execute thecommand on. As a result, the gesture application will qualify thecommand, “Rotate Object,” with the recognized qualifiers specifying toexecute the command on “Object 1” and to “Rotate by 30 degrees” toobtain the qualified command “Rotate Object 1 by 30 Degrees.”

FIG. 3 illustrates a camera 310 detecting a user's 300 hand gestureinput 350 and a microphone 340 (input device) detecting a user'snon-hand gesture input 360 according to an embodiment of the invention.As noted above, a camera 310 can be instructed by a gesture applicationto detect and capture hand gesture inputs 350 and/or non-hand gestureinputs 360 made by the user 300.

As illustrated in FIG. 3, in one embodiment, the camera 310 captures theuser 300 making a hand gesture 350 with both hands. Additionally, asillustrated in FIG. 3, in the present embodiment, the hand gesture 350made by the user 300 involves the user moving both of his hands awayfrom one another following a diagonal motion. As noted above, the camera310 will capture the motion. Further, as illustrated in FIG. 3, thecamera 310 can additionally capture a shape of the hand gesture 350 madeby the user 300. The camera 310 will capture the user 300 having indexfingers and thumbs on both hands extended, while the remaining figuresare retracted or dosed. In other embodiments, the shape of the handgesture 350 made by the user 300 includes additional fingers, extensionsof the fingers, retractions of the fingers, and/or configurations of thefingers.

As noted above, the gesture application will analyze the hand gestureinput 350 to identify a corresponding command. Additionally, inanalyzing the captured hand gesture input 350, the gesture applicationwill compare the number of hands used, the shape of the hands, and/ormotions made by the hands to recognized hand gestures. Further, as notedabove, the recognized hand gestures include commands that correspond toa number of hands used, hand shapes made, and/or hand motions made.

In one embodiment, the gesture application will scan a list ofrecognized hand gestures or access a database of recognized commands andrecognized qualifiers for a corresponding command that involves the userusing both hands, with index fingers and thumbs extended (the shape ofthe hands), and moving away from one another in a diagonal movement(motions of the hands). In one embodiment, the gesture application willfind that the number of hands, shape of the hands, and the motion ofhands match a criteria of a “zoom out” command. As a result, the gestureapplication will choose the “zoom out” command to execute.

Further, as illustrated in FIG. 3, one or more input devices, in thepresent embodiment, a microphone 340, can be instructed by the gestureapplication to detect and capture one or more non-hand gesture inputs360. As noted above, in one embodiment, one or more input devices (themicrophone 340) can detect the non-hand gesture input 360 while thecamera 310 concurrently captures the hand gesture 350. In anotherembodiment, the microphone 340 detects the non-hand gesture input 360sequentially after the camera 310 has detected and captured the handgesture 350.

Further, as noted above and illustrated in FIG. 3, the non-hand gestureinput 360 may include speech. Additionally, as noted above, the non-handgesture input 360 is used to qualify the command made from the handgesture 350 by specifying a degree of the command. As illustrated inFIG. 3, the microphone 340 will detect and capture the non-hand gestureinput 360 as “3 times,” The gesture application will then searchrecognized qualifiers corresponding to the command “zoom out” todetermine if the speech qualifier is recognized and whether the speechqualifier is compatible with the “zoom out” command. In on embodiment,the gesture application will find a recognized qualifier that includesspeech to qualify the “zoom out” command by performing it “n times.” Asa result, the gesture application will qualify the command of “zoom out”with the qualifier “3 times” and proceed to perform the qualifiedcommand of “zoom out 3 times.”

FIG. 4 illustrates users 410, 420, and 430 entering hand gesture inputsand non-hand gesture inputs to qualify commands that will be executed onobjects shown on a display device according to an embodiment of theinvention. As noted above and illustrated in FIG. 4, in one embodiment,a machine includes one or more cameras 450 and/or input devices 460configured by a gesture application to capture one or more hand gestureinputs and/or one or more non-hand gesture inputs from users.Additionally as noted above, in one embodiment, one or more cameras 455can be worn by a user 1 410 as an accessory.

Further, as shown in FIG. 4, in one embodiment, a display device 440displays multiple objects 470, 480, and 490. The display device 440 isany device that can display or project objects and/or instructions. Inone embodiment, the display device is a monitor or television. In otherembodiments, the display device 440 is a projector. As noted above, oneor more objects can be displayed on the display device 440 and can beinteracted with by users entering hand gesture inputs and non-handgesture inputs.

As illustrated in FIG. 4, in one embodiment, user 1 410 is wearingcamera helmet 455 and the camera helmet 455 is capturing a non-handgesture input, gaze qualifier 1 416. Additionally, while the camerahelmet 455 is capturing gaze qualifier 1 with a line of sight on object480, cameras 450 are capturing the hand gesture 1 input 413 by user 1410. In another embodiment, the camera helmet 455 can also be utilizedby user 1 410 to capture the hand gesture 1 input 413. Further, asillustrated in FIG. 4, user 1 410 is also concurrently entering anothernon-hand gesture input, speech qualifier 1 419. Speech qualifier 1 419will be captured by input device 460, a microphone, for the gestureapplication. As illustrated in FIG. 4, speech qualifier 1 419 includesthe “rotate” command. In one embodiment, the gesture application canutilize the command included in the speech qualifier to reinforce orreconfirm the hand gesture command. By analyzing hand gesture 1 input413, gaze qualifier 1 416, and speech qualifier 1 419, the gestureapplication determines that user 1 410 wishes to enter the qualifiedcommand, “rotate object 480 60 degrees.”

Additionally, while the gesture application is qualifying the commandfor user 410, the gesture application can also qualify a command foruser 2 420. As illustrated in FIG. 4, one or both of the cameras 450 arecapturing a hand gesture 2 input 423 and a gaze qualifier 2 426 fromuser 2 420. Additionally, the microphone 460 will also be capturing anon-hand gesture input, speech qualifier 2 429. The gesture applicationwill determine from hand gesture 2 input 423 that a panning action fromright to left can be executed. Additionally, the gesture applicationwill qualify the command with gaze qualifier 2 426 to identify that thecommand will be executed on object 470 and speech qualifier 429 to panto the edge of the screen. As a result, the gesture application willdetermine that user 2 420 is trying to “pan object 470 to the edge ofthe screen.”

Further, in one embodiment, the gesture application can qualify acommand for user 3 430 after executing the qualified commands for user 1410 and user 2 420. As illustrated in FIG. 4, one or more of the cameras450 can capture hand gesture 3 input 433 and non-hand gesture input 3,the gaze qualifier 436 from user 3 430. Further, the microphone 460 cancapture another non-hand gesture input, speech qualifies 3, 439. As aresult, the gesture application will determine that user 3 430 is tryingto “delete object 490.”

As noted above, in embodiment, if the gesture application can notidentify a command that corresponds to the hand gesture, the gestureapplication can instruct the display device 440 to output a notrecognized message and/or a message prompting the user to redo the handgesture. In another embodiment, the gesture application willadditionally instruct the display device 440 to display an example orsample hand shapes and/or motions that are recognized by the gestureapplication. The gesture application can also do this when the gestureapplication cannot identify one or more of the non-hand gesture inputs.

FIG. 5 illustrates a machine with an embedded gesture application and agesture application 510 stored on a removable medium being accessed bythe machine 500 according to an embodiment of the invention. For thepurposes of this description, a removable medium is any tangibleapparatus that contains, stores, communicates, or transports theapplication for use by or in connection with the machine 500. As notedabove, in one embodiment, the gesture application 510 is firmware thatis embedded into one or more components of the machine 500 as ROM. Inother embodiments, the gesture application 510 is a software applicationwhich is stored and accessed from a hard drive, a compact disc, a flashdisk, a network drive or any other form of computer readable medium thatis coupled to the machine 500.

FIG. 6 is a flow chart illustrating a method for executing a qualifiedcommand according to an embodiment of the invention. The method of FIG.6 uses one or more cameras and/or one or more input devices to detectone or more hand inputs and one or more non-hand gesture inputs.Additionally, the method of FIG. 6 utilizes a gesture application thatwill recognize the corresponding hand gesture input and qualify acommand corresponding to the hand gesture input with one or morequalifiers. In other embodiments, the method of FIG. 6 uses additionalcomponents and/or devices in addition to and/or in lieu of those notedabove and illustrated in FIGS. 1, 2, 3, 4, and 5.

As noted above, the gesture application will initially instruct one ormore cameras and/or one or more input devices to detect a hand gestureinput for identifying a command 600. As noted above, in detecting thehand gesture input, one or more cameras and/or one or more input deviceswill capture a user making a hand gesture input. Once the hand gestureinput has been captured, the gesture application will compare thecapture hand gesture input to recognized hand gestures stored on themachine. As noted above, in one embodiment, the recognized hand gesturesare stored as a list in a file. In another embodiment, the recognizedhand gestures are stored in a database accessible to the machine.

Further, as noted above, if the gesture application does not recognizethe hand gesture captured, the gesture application can instruct adisplay device coupled to the machine to output a message indicatingthat the hand gesture input was not recognized and/or for the user tore-input one or more hand gesture inputs. Once the gesture applicationhas identified a command corresponding to the hand gesture input, thegesture application will proceed to detect one or more non-hand gestureinputs to qualify the command 610.

In another embodiment, one or more non-hand gesture inputs may have beencaptured by one or more input devices before the hand gesture input wasis captured and identified. In other embodiments, one or more handgesture inputs and one or more non-hand gesture inputs can be capturedconcurrently with one or more cameras and/or one or more input devices.As noted above, in one embodiment, one or more non-hand gesture inputsinclude speech and/or gaze.

Once one or more non-hand gesture input has been captured, the gestureapplication will then proceed to qualify the command with one or more ofthe recognized qualifiers. As noted above, the recognized qualifierscorrespond to the identified command and specify parameters for thecommand. Similar to above, in one embodiment, the recognized qualifiersare stored as a list in a file. In another embodiment, the recognizedqualifiers are stored in a database accessible to the machine. Thegesture application will scan the recognized qualifiers for one thatmatches one or more non-hand gesture inputs that have been captured.

Additionally, if the gesture application does not recognize the non-handgesture input, the gesture application can instruct the display devicecoupled to the machine to output a message indicating that the non-handgesture input was not recognized and/or for the user to re-input one ormore non-hand gesture inputs. In another embodiment, the gestureapplication can instruct the display device to display a list ofrecognized qualifiers corresponding to the command.

Once the gesture application has qualified the command, the gestureapplication will configure a processor to execute a qualified command ona machine in response to the hand gesture input and one or more non-handgesture inputs 630. The process is then complete or the gestureapplication continues to instruct one or more cameras and/or one or moreinput devices to detect one or more hand gesture inputs and one or morenon-hand gesture inputs to act as qualifiers by repeating the methoddisclosed in FIG. 6. In other embodiments, the method of FIG. 6 includesadditional steps in addition to and/or in lieu of those depicted in FIG.6.

FIG. 7 is a flow chart illustrating a method for executing a qualifiedcommand according to another embodiment of the invention. Similar to themethod of FIG. 6, the method of FIG. 7 uses one or more cameras and/orone or more input devices to detect one or more hand gesture inputs andone or more non-hand gesture inputs. Additionally, the method of FIG. 7utilizes a gesture application that will recognize the correspondinghand gesture input and qualify a command corresponding to the handgesture input with one or more recognized qualifiers. In otherembodiments, the method of FIG. 7 uses additional components and/ordevices in addition to and/or in lieu of those noted above andillustrated in FIGS. 1, 2, 3, 4, and 5.

As noted above, in one embodiment, the gesture application initiallyidentifies a user which is entering commands to be executed 700. Asnoted above, the gesture application identifies a user with one or morecameras and/or one or more input devices 705. In one embodiment, thegesture application instructs a camera and/or one or more input devicesto capture hand gestures, speech, and/or gaze to identify the user 710.By identifying the user, the gesture application can determine whetherto grant the user access to the machine and whether to execute qualifiedcommands entered by the user. The gesture application can utilizerecognized hand gestures, gazes, and/or voices that correspond to a userto identify the user. In another embodiment, the gesture application canadditionally instruct one or more cameras and/or one or more inputdevices to begin to detect hand gestures inputs and non hand gestureinputs from the user 715 while identifying the user.

In other embodiments, the gesture application can skip identifying theuser and immediately instruct one or more cameras and/or one or moreinput devices to begin to detect hand gesture inputs from a user 715. Asnoted above, in detecting the hand gesture inputs, the gestureapplication uses one or more cameras and/or one or more input devices tocapture a number of shapes used, a shape of one or more of the hands,and/or a motion of one or more of the hands.

After the gesture application has captured one or more hand gestureinputs by the user, the gesture application compares the captured handgesture input to recognized hand gestures 720. As noted above, therecognized hand gestures can be stored as a file on the machine or as adatabase accessible to the machine. The gesture application will thenattempt to identify a command corresponding to the hand gesture input725 by comparing a number of hands used, shapes of the hands, and/ormotions made by the hands from the hand gesture input to recognized handgestures.

In one embodiment, if the gesture application determines that the handgesture was not recognized, the gesture application will instruct adisplay device coupled to the machine to output a not recognized message730 and/or a message prompting the user to redo the hand gesture. Inanother embodiment, the gesture application will additionally instructthe display device to display an example or sample hand shapes and/ormotions that are recognized by the gesture application.

In other embodiments, if the gesture application finds a matchingrecognized hand gestures, the gesture application can select a commandcorresponding to the matching recognized hand gesture and proceed toattempt to qualify the command with one or more non-hand gesture inputs.As noted above, in one embodiment, one or more qualifiers are speechinputs and/or gaze inputs. Additionally, the qualifiers specifyparameters of the command. In one embodiment, the qualifiers can specifydegrees for the command, such as how many times to zoom in/out or howmuch to increase/decrease a volume. In another embodiment, thequalifiers specify which object on the display device to execute thecommand on. In other embodiments, one or more qualifiers can reinforcethe command. In qualifying the command, the gesture application promptsone or more cameras and/or one or more input devices to detect one ormore non-hand gesture inputs 740.

Once the gesture application has captured one or more non-hand gestureinputs, the gesture application can compare the captured non-handgesture qualifiers to recognized qualifiers corresponding to theidentified command 750. The gesture application can then determinewhether any of the recognized qualifiers match the non-hand gestureinputs 755. Similar to above, if the gesture application can not cannotfind a match, the gesture application can output a not recognizedmessage 760 and/or a message for the user to re-input a non-hand gestureinput. Additionally, in another embodiment, the gesture application caninstruct the display device to output the recognized qualifiers thatcorrespond to the identified command 770. The gesture application maythen instruct one or more cameras and/or one or more input devices todetect one or more non-hand gesture qualifiers 740 in order to find oneor more matching qualifiers.

Once the gesture application has found matching recognized qualifiersfor the non-hand gesture inputs, the gesture application can proceed toconfigure a processor to execute a qualified command on the machine inresponse to the hand gesture input and one or more non-hand gestureinputs 790. Additionally, if one or more non-hand gesture inputs includea gaze input, the gesture application can also determine whether one ormore of the non-hand gesture qualifiers specify an object to execute thecommand on 780.

Once the gesture application has identified the object to execute thecommand on, the gesture application can proceed to configure a processorto execute a qualified command on the machine in response to the handgesture input and one or more non-hand gesture inputs 790. The processis then complete or the gesture application continues to instruct one ormore cameras and/or one or more input devices to detect one or more handgesture inputs and one or more non-hand gesture inputs to act asqualifiers by repeating the method disclosed in FIG. 7. In oneembodiment, the method of FIG. 7 includes additional steps in additionto and/or in lieu of those depicted in FIG. 7.

By identifying commands associated with hand gesture inputs andidentifying qualifiers corresponding to the command with non-handgesture inputs, a user can have accurate and precise control overqualified commands that a machine executes. Additionally, by identifyinga user with one or more hand gesture inputs and/or one or more non-handgesture inputs, security, user friendliness, and convenience are gainedby the user.

1. A method for executing a qualified command comprising: detecting a hand gesture input for identifying a command; detecting one or more non-hand gesture inputs to qualify the command; and configuring a processor to execute the qualified command on a machine in response to the hand gesture input and one or more non-hand gesture inputs.
 2. The method for executing a command of claim 1 wherein a camera or one or more input devices captures the hand gesture input while the camera or one or more input devices concurrently captures one or more non-hand gesture inputs.
 3. The method for executing a command of claim 2 further comprising comparing the captured hand gesture input to recognized hand gestures and comparing one or more non-hand gesture inputs to recognized qualifiers.
 4. The method for executing a command of claim 1 wherein a camera detects and captures the hand gesture input while one or more input devices concurrently detects and captures one or more non-hand gesture inputs.
 5. The method for executing a command of claim 1 wherein one or more non-hand gesture inputs are used to define a parameter for the command.
 6. The method for executing a command of claim 1 wherein one or more non-hand gesture inputs are used to identify an object to execute the command on.
 7. A machine for executing a qualified command comprising: a processor; one or more cameras configured to capture a hand gesture input; one or more input devices configured to capture one or more non-hand gesture inputs; and a gesture application executed by the processor and configured to identify a command corresponding to the hand gesture input and qualify the command in response to one or more non-hand gesture inputs.
 8. The machine of claim 7 further comprising a display device configured to display a list of recognized qualifiers corresponding to the command.
 9. The machine of claim 7 wherein one or more input devices may include at least one from the group consisting of a proximity sensor, a biometric device, one or more cameras, a microphone, a touch screen, and a mouse.
 10. The machine of claim 8 wherein the display device is additionally configured to output a visual message on the display device when the hand gesture or one or more non-hand gestures are not recognized by the gesture application.
 11. The machine of claim 7 further comprising a database of recognized hand gestures and recognized qualifiers corresponding to the recognized hand gestures.
 12. A computer-readable program in a computer-readable medium comprising: a gesture application configured to identify a command corresponding to one or more hand gestures; wherein the gesture application is further configured to qualify the command with one or more speech or gaze qualifiers; and wherein the gesture application is additionally configured to execute a qualified command from recognized commands stored on a storage device in response to one or more hand gestures and one or more speech or gaze qualifiers.
 13. The computer-readable program in a computer-readable medium of claim 12 wherein the gesture application is additionally configured to identify one or more objects on a display device to execute the qualified command on in response to one or more speech or gaze qualifiers.
 14. The computer-readable program in a computer-readable medium of claim 12 wherein the gesture application is further configured to specify a degree of the qualified command in response to one or more speech or gaze qualifiers.
 15. The computer-readable program in a computer-readable medium of claim 12 wherein the gesture application is further configured to identify a user with one or more hand gestures or one or more speech or gaze qualifiers. 